[FLINK-38857][Model] Introduce a Triton inference module under flink-models #27385
+2,429
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
davidradl
reviewed
Jan 5, 2026
davidradl
reviewed
Jan 5, 2026
davidradl
reviewed
Jan 5, 2026
davidradl
reviewed
Jan 5, 2026
github-actions
bot
added
the
community-reviewed
davidradl
reviewed
Jan 7, 2026
davidradl
reviewed
Jan 7, 2026
davidradl
reviewed
Jan 7, 2026
featzhang
changed the title
featzhang
force-pushed
the
FLINK-38857
branch
from
d579334 to
6b3d20d
Compare
dianfu
reviewed
Jan 30, 2026
...ink-model-triton/src/main/java/org/apache/flink/model/triton/TritonModelProviderFactory.java
Outdated
Show resolved
Hide resolved
...nk-model-triton/src/main/java/org/apache/flink/model/triton/AbstractTritonModelFunction.java
Outdated
Show resolved
Hide resolved
...nk-model-triton/src/main/java/org/apache/flink/model/triton/AbstractTritonModelFunction.java
Show resolved
Hide resolved
...k-model-triton/src/main/java/org/apache/flink/model/triton/TritonInferenceModelFunction.java
Outdated
Show resolved
Hide resolved
...k-model-triton/src/main/java/org/apache/flink/model/triton/TritonInferenceModelFunction.java
Outdated
Show resolved
Hide resolved
...nk-model-triton/src/main/java/org/apache/flink/model/triton/AbstractTritonModelFunction.java
Outdated
Show resolved
Hide resolved
flink-models/flink-model-triton/src/main/java/org/apache/flink/model/triton/TritonUtils.java
Outdated
Show resolved
Hide resolved
flink-models/flink-model-triton/src/main/java/org/apache/flink/model/triton/TritonUtils.java
Outdated
Show resolved
Hide resolved
...nk-model-triton/src/main/java/org/apache/flink/model/triton/AbstractTritonModelFunction.java
Outdated
Show resolved
Hide resolved
...nk-model-triton/src/main/java/org/apache/flink/model/triton/AbstractTritonModelFunction.java
Outdated
Show resolved
Hide resolved
davidradl
reviewed
Jan 30, 2026
| .withDescription( | ||
| Description.builder() | ||
| .text( | ||
| "Reserved for future use (v2+). Currently has NO effect in v1. " |
Contributor
There was a problem hiding this comment.
I am curious - what is the benefit of adding something we do not use yet? It would seem simpler to add it when we use it.
davidradl
reviewed
Jan 30, 2026
| public static final ConfigOption<Long> TIMEOUT = | ||
| ConfigOptions.key("timeout") | ||
| .longType() | ||
| .defaultValue(30000L) |
Contributor
There was a problem hiding this comment.
we should use the Duration type for this
davidradl
reviewed
Jan 30, 2026
| Description.builder() | ||
| .text( | ||
| "Full URL of the Triton Inference Server endpoint, e.g., %s", | ||
| code("http://localhost:8000/v2/models")) |
Contributor
There was a problem hiding this comment.
worrying that the example is http i.e. unsecure.
davidradl
reviewed
Jan 30, 2026
| .booleanType() | ||
| .defaultValue(false) | ||
| .withDescription( | ||
| "Whether this is the start of a sequence for stateful models."); |
Contributor
There was a problem hiding this comment.
it would be useful to have a reference as to what a sequence is in this context
davidradl
reviewed
Jan 30, 2026
| ConfigOptions.key("compression") | ||
| .stringType() | ||
| .noDefaultValue() | ||
| .withDescription("Compression algorithm to use (e.g., 'gzip')."); |
Contributor
There was a problem hiding this comment.
can we give a reference or list the valid values for this?
davidradl
reviewed
Jan 30, 2026
|
|
||
| public static final ConfigOption<String> CUSTOM_HEADERS = | ||
| ConfigOptions.key("custom-headers") | ||
| .stringType() |
Contributor
There was a problem hiding this comment.
Headers are MAP <STRING, ARRAY>. Can we avoid using json and use the standard Flink map type passing lists of strings in the way that list types would expect?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose of this change
This PR introduces an optional Triton-based inference module under
flink-models, enabling Apache Flink to invoke NVIDIA Triton Inference Server for model inference.The integration is implemented at the runtime layer via the existing model provider SPI, allowing users to define Triton-backed models using
CREATE MODELand execute inference throughML_PREDICT, without requiring any changes to the planner, runtime semantics, or SQL behavior.This module is designed as a reusable and extensible foundation for integrating external inference services into Flink’s model inference framework.
Summary of changes
flink-model-tritonmodule underflink-modelsVerification
Impact assessment
This change is fully optional and isolated under
flink-models.Documentation
docs/connectors/models/triton.mdRelated issues